Culture in the Classroom: Challenges for Assessment in Ill-Defined Domains

While cognitive tutoring has been successful in well-defined domains, much less work has been completed in cognitive tutoring in ill-defined domains. Skill assessment is one of the difficult challenges that that must be solved before cognitive tutoring can become a reality in these domains. Assessment formats commonly used by instructors in ill-defined domains do not always have a correct answer or follow a finite set of solution paths. This necessitates clever ways of evaluating student responses to open-ended questions in which variability in both approach and final solution is a given. Additionally, there may be difficulty in covering the full problem space with a single assessment. In this paper we describe complementary assessments developed to evaluate a tutoring system in a classroom study. Critical analysis questions which are easily machine-interpretable and writing samples that address more nuanced understanding assess different skills that will be incorporated into a set of intelligent tutors that cover the cultural learning domain.